A well-evaluated state covariance matrix avoids error propagation due to divergence issues and, thereby, it is crucial for a successful state estimator design. In this paper we investigate the performance of the state covariance matrices used in three unconstrained Extended Kalman Filter (EKF) formulations and one constrained EKF formulation (CEKF). As benchmark case studies we have chosen: a) a batch chemical reactor with reversible reactions whose system model and measurement are such that multiple states satisfy the equilibrium condition and b) a CSTR with exothermic irreversible reactions and cooling jacket energy balance whose nonlinear behavior includes multiple steady-states and limit cycles. The results have shown that CEKF is in general the best choice of EKF formulations (even if they are constrained with an ad hoc clipping strategy which avoids undesired states) for such case studies. Contrary to a clipped EKF formulation, CEKF incorporates constraints into an optimization problem, which minimizes the noise in a least square sense preventing a bad noise distribution. It is also shown that, although the Moving Horizon Estimation (MHE) provides greater robustness to a poor guess of the initial state, converging in less steps to the actual states, it is not justified for our examples due to the high additional computational effort.